Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrates GitLab Runner into DTaaS #1082

Closed
wants to merge 24 commits into from
Closed

Integrates GitLab Runner into DTaaS #1082

wants to merge 24 commits into from

Conversation

aryanpingle
Copy link
Contributor

Integrates GitLab Runner into DTaaS

Addresses #892 by integrating a GitLab Runner setup into the DTaaS software.

Type of Change

✔ New feature
✔ Documentation update

Description

Adds a compose.runner.yml file that spins up a Docker container running the gitlab/gitlab-runner image, and automatically starts a gitlab runner based on the settings in config.toml.

Testing

Tested using a localhost and a server installation, both using a server-hosted gitlab instance.

Impact

The process of registering a GitLab Runner will be greatly simplified. Developers will be able to use the digital twins preview page with a self-hosted runner.

Additional Information

  • On localhost, the hello-world digital twin fails during execution as the runner executor cannot access the host machine's localhost.
  • Currently, network_mode is set to host for the gitlab-runner container. This should instead use or create a docker network.
  • Added a guide in the services/gitlab directory.
  • Need to update/create the admin guide for using the Digital Twins Preview Page.

Checklist

  • My code adheres to the coding and style guidelines of the project.
  • I have added tests for all the new code and any changes made to
    existing code.
  • I have made corresponding changes to the documentation.

prasadtalasila
prasadtalasila reviewed Nov 26, 2024
View reviewed changes
Copy link
Contributor

@prasadtalasila prasadtalasila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aryanpingle Thanks for the PR. Please see the comments.

deploy/services/gitlab/compose.runner.yml Outdated Show resolved Hide resolved
deploy/services/gitlab/gitlab-runner/config.toml Outdated
token = "xxx"
executor = "docker"
[runners.docker]
tls_verify = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this solve the problem of self-generated certificates?

deploy/services/gitlab/GITLAB-RUNNER.md Outdated Show resolved Hide resolved
@aryanpingle
Copy link
Contributor Author

The gitlab-runner container is built on the gitlab/gitlab-runner image using alpine and contains the gitlab-runner executable. Starting the container executes gitlab-runner register and gitlab-runner start automatically, so we only need to deal with the runner-config.toml file to set things up.

The gitlab/gitlab-runner image uses a custom image of its own, from the "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper" repository. This uses the docker engine to spawn a sibling container to gitlab-runner, which actually executes the CI/CD job.

Now, if you try using docker ps you won't see the sibling executor container/s, for a reason I don't fully understand. Here's a link to the SO answer: https://stackoverflow.com/a/41994792/8089674.

@aryanpingle aryanpingle force-pushed the feature/integrated-runner branch from a583833 to 721271f Compare December 10, 2024 11:59
@prasadtalasila
Copy link
Contributor

@aryanpingle A related correction in the docs

line-69 of deploy/services/gitlab/INTEGRATION.md file should be:

If the DTaaS application is hosted at https://foo.com/, then configure the following files:

@prasadtalasila
Copy link
Contributor

The gitlab-runner container is built on the gitlab/gitlab-runner image using alpine and contains the gitlab-runner executable. Starting the container executes gitlab-runner register and gitlab-runner start automatically, so we only need to deal with the runner-config.toml file to set things up.

The gitlab/gitlab-runner image uses a custom image of its own, from the "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper" repository. This uses the docker engine to spawn a sibling container to gitlab-runner, which actually executes the CI/CD job.

Now, if you try using docker ps you won't see the sibling executor container/s, for a reason I don't fully understand. Here's a link to the SO answer: https://stackoverflow.com/a/41994792/8089674.

please add this information in a separate section at the end of GITLAB-RUNNER.md

@prasadtalasila prasadtalasila added this to the Release v0.7.0 milestone Dec 13, 2024
@aryanpingle
Copy link
Contributor Author

The latest commits add content from Chapter 3, Section 3 and Chapter 4 from "Design and implementation of an integrated DevOps framework for Digital Twins as a Service software platform" (Scherma V.) as a new section "DevOps Framework" in the developer documentation. I've reordered a few sections and edited some parts to make it more developer friendly.

prasadtalasila
prasadtalasila reviewed Dec 17, 2024
View reviewed changes
Copy link
Contributor

@prasadtalasila prasadtalasila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aryanpingle thanks for the updates. please see the comments.

docs/developer/devops/api.md Outdated Show resolved Hide resolved
docs/developer/devops/api.md Outdated Show resolved Hide resolved
docs/developer/devops/gitlab-ci-cd.md Show resolved Hide resolved
docs/developer/devops/implementation.md Outdated Show resolved Hide resolved
docs/developer/devops/implementation.md Show resolved Hide resolved
@prasadtalasila
Copy link
Contributor

@aryanpingle please add the following citation at the end of pages taken from Vanessa's thesis:

Ref: Vanessa Scherma, Design and implementation of an integrated DevOps framework for Digital Twins as a Service software platform, Master’s Degree Thesis, Politecnico Di Torino, 2024.

@prasadtalasila prasadtalasila marked this pull request as ready for review December 17, 2024 13:43
@prasadtalasila prasadtalasila mentioned this pull request Dec 19, 2024
Merged
prasadtalasila added a commit that referenced this pull request Dec 19, 2024
@prasadtalasila
Integrates GitLab Runner and updates documentation (#1115 and #1082)
2257ef5 
  - Adds integrated gitlab runner to all installation scenarios.
    It is now possible to have one runner that can run all
    the jobs sent by DevOps requests.
  - Updates DevOps documentation and ports corrections from
    documentation of release v0.6.0

---------
Co-authored-by: aryanpingle <[email protected]>
@prasadtalasila
Copy link
Contributor

Thanks @aryanpingle. The changes have been merged via PR #1115.

This was referenced Dec 19, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prasadtalasila prasadtalasila prasadtalasila left review comments

Assignees
No one assigned
Labels
None yet
Projects
Digital Twin as a Service
Status: Done
Milestone
Release v0.7.0
Development

Successfully merging this pull request may close these issues.
2 participants
@aryanpingle @prasadtalasila